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ABSTRACT 

Much of the intrinsic wealth of planning and 
instructional information available from achievement testing programs 
goes untapped in typical reporting procedures. Large-scale programs 
reporting only pupil scores and the results of aggregating those 
scores stop far short of the purposes intended and fail to realize 
the potential of such information. Providing a rationale for and a 
description of user-oriented reports and reporting procedures, the 
discussion focuses on individual pupil performance, class analyses 
for teachers and counselors, grade-within-school performance for 
principals and school staff, and system- wide grade performance for 
central administrative staff. The models used could lead to greater 
understanding, utilization and implementation of the data. 
(Author) 
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LARGE-SCALE STANDARDIZED TESTING PROGRAMS — 
NEW VISTAS IN USER-ORIENTED REPORTING 



Jules Grosswald, Director 
Division of Testing Services 
Board of Education Bldg. - Rm. ^07 
21st Street and The Parkway 
Philadelphia, Pa. 19103 



Jules Grosswald 
Division of Testing Services 
The School District of Philadelphia 



SUMMARY 



Ob ject ives 



Much of the intrinsic wealth of planning and instructional information 
available from testing programs involving standardized achievement tests goes 
untapped in typical reporting procedures. Large-scale testing programs which 
report only the test scores of pupils and the results of aggregating those 
scores stop far short of the purposes intended by the school system for such 
programs and fail to realize the availability or potential of such information. 

This paper is intended to provide a rationale for and a description of 
user-oriented reports and reporting procedures. The models used for test-score 
and item performance reporting could lead to greater understanding, utilization, 
and implementation of the school system's testing program data. 

Perspectives or Theoretical Framework 

The major purpose of standardized achievement tests is to provide school 
personnel with objective and dependable data which can serve as a basis for the 
evaluation of individual, class and entire grade achievement levels in each of 
the basic skills. In order to accomplish this purpose, scores are reported for 
individual pupils; averages are reported for every class at each grade level 
and averages are reported for every grade tested in each school. In addition, 
averages for school clusters, for the system at large, and national norms at 
every grade level make it possible to compare individual and group performance 
with different reference populations. These data are designed to help the 
school administrator identify areas of strength and weakness in the instructional 
program so that programmatic changes might be directed towards those skills that 
are in need of greater emphasis, of increased "drive 11 by teachers and supervisors 
of instruction and o* increased attention on the part of those who are involved 
with curriculum improvement and reorganization. 

In addition, the testing program data affords the administrative staff 
with the basis for a variety of decisions dealing with instructional planning, 
programming, organization and materials selection as well as resource planning 
and al location. 

Although the data base is the same, the selection and formating of data 
for decision-making use varies. 

The literature suggests that very little in the way of change or modifica- 
tion of test data for individual reporting or aggregate reporting has occurred. 
There is the need for some new perspectives related to data display and data 
use. 
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The advent of highly efficient, sophisticated computing programs and 
equipment now make possible data aqgregation and analyses that could not be 
considered in the past. As a result, many users are not ?ware of possibilities 
for new approaches in test data applications. 

Methods and/or Techniques 

The Performance Analysis Record 

This record has been developed to provide for pupil, teacher, parent, 
counselor, etc. an individual report of pupil test performance. Prepared 
individually for each pupil in triplicate, the record provides converted 
scores, national percentile ranks of those scores; a section analyzing pupil 
performance in each subtest giving the percent of items correct, the percent 
of items incorrect, and percent of items not answered; and a section showing 
item by item grouped according to the publisher^ skMIs classification 
whether the i tern was answered correctly, incorrectly or omitted. (S^e Sample I.) 

The individual records are aggregate for school and system use. 



The Summary Tables 

These tables organize the aggregate data by areas tested and provide a 
school performance distribution based on given ranges of national percentile 
ranks, and show the relationship of averages to system and national norms in 
several different contexts. 

The tables are provided for several administrative levels — school and 
larger administrative units. ( See Sample 2.) 



Item Analysis Summaries 
For each subtest of the battery, the following information is provided: 

(1) the numbers of the items in the subtest for which data are given, 

(2) the percents of pupils in the grade in the school who completed 

the entire batter> and who answered each item correctly, 

(3) the percents of pupils in the publisher's standardization population 

who completed the entire battery and who answered each item 
correctly, and 

(4) the percents of pupils in the grade in the city who completed the 

entire battery and who answered each item correctly. 

These summaries are provided for larger administrative units as well. 
(See Sample 3.,) 
Data Source 

The models, format of data, and applications arc a result of several years 
of development and implementation In (he School Distilcl of Phi f.iclHphl.i. 
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SAMPLE 2 



Tlil SCHOOL DI -.TRICT OF PHILADELPHIA 
Office of Research and Evaluat ! on 
Division of Testing Services 

I973-/A PHILADELPHIA CITY-WIDE TESTING PROGRAM 

STANFORD EARLY SCHOOL ACHIEVEMENT TEST - LEVEL I 

SUMMARY TABLES FO* MAY U74 ADMINISTRATION - KINDERGARTEN 

SCH00L SAMPLE DISTRICT NUMBER 9 NUMBER TESTED 122 



TABLE 1: SCHOOL PERFORMANCE DISTRIBUTION 



TEST AREA 




PERCENTAGE OF PUPILS SCORING 


Below Nat'1 
16th %ile 


Between N. 
16th to 49th 


st'l Siles 

: 50th to 84th 


Nat'l 85th 
%\ le or above 


Envi ronment 


20 


35 


: 32 


13 


Mathemat i cs 


19 


31 


! 33 


17 


Letters and Sounds 


8 


27 


34 


31 


Aural Comprehension 


16 


31 


39 


14 


Total 


15 


3i : 


* 


19 



TABLE 2: RELATIONSHIP OF SCHOOL AVERAGES TO NORMS 



TEST AREA 


%\ le Rank Based 
on Nat'l Pupil 
Norms 


%\\e Rank Based 
on Phi la. Bldg. 
Norms 


PERCENTAGE OF 
PUPILS SCORING 


At or above 
Nat'l Median 


Envi ronment 


36 


41 


45 


Mathematics 


44 


52 


50 


Letters and Sounds 


62 


69 


65 


Aural Comp rehens i on 


50 


57 


1 

53 


Total 


48 


51 


54 



T 74-124-S 
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SAMPLE 3 
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LEVEL S - GRADE 7 

TEST R: READING COMPREHENSION 

Skills Classification 

Ths skills tested if Test R may be classed under four headings: dstails, 
purpose, organisation, and evaluation* For each of these four classes, then, 
the items concerned primarily with the e kills involved in that class will be 
identified. (Many of theee iteme could be readily clsssifisd in more than one 
way; hence, the classification is somewhat subjective.) 

Skills: D (Dstails) - To Recognize and Understand Stated or Implied Factual 
Dstails end Relationships 
D-l To recognize and understand important facts and dstails 
D-2 To recognize and understandj^plied facts and rslationships 
D-3 To deduce the meaning of worde or phrases from context 



Work Table 

Skill D-l: To recognize and understand important facts and details 



Item 

No. 



Skill 
Measured 



Difficulty 
of 

Item 



Percent of Pupils 
Answering Correctly 



Phila. 



NatU 



School 



Difference 
(Nat'l 4 School 
■f or 



1 
21 
26 
29 
33 
44 

45 
47 
50 
58 

59 
62 

65 
68 

70 
71 
74 
2L 



D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 
D-l 



E 
A 
A 
A 
A 
A 
A 
A 
A 
E 
A 
A 
A 
A 
A 
A 
A 
H 



it 



32 



IT 



-Si. 



il 



33 



-21 



11 



38 



-2SL 



Jl 



57 



-21 



55 



40 



81 

13" 



i 



2E 



5 



26 

T3" 



37 



-21 



21 



- 4 



T 



3ET 



T 

S5r 



-15» 



Number of Items 


- 18 


Sum of ♦ Differences 


-+ 1 


Sum of - Differences 


--191 


Algebraic Sum 


--190 


Algebraic Sum 


— - -11 


_ A W»ff - 18, 





H • Hard; A • Average; 8 - Easy 
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* Denotes Items having a difference between national and school percentages 
greater than the average of Item difficulty differences for the school. 
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TABLE 2 
LEVEL S - GRADE 7 



Work Table 

Skill D-2t To recognize and understand implied facts and relationships 



Difference I 
(Mst'l 4 School ) 
♦ or 







Dlfficu] 


Item 


Skip 


of 


No, 


Measured 


Its* 


2 


D-2 


S 


3 


D-2 


A 


4 


D-2 


A 


5 


D-2 


E 


7 


D-2 


£ 


8 


D-2 


A 


10 


D-2 


A 


17 


D-2 


H 


19 


D-2 


A 


20 


D-2 


A 


22 


D-2 


£ 


23 


D-2 


A 


24 


D-2 


' £ 


25 


D-2 


A 


27 


D-2 


£ 


28 


D-2 


£ 


31 


D-2 


A 


32 


D-2 


£ 


37 


D-2 


A 


46 


D-2 


A 


48 


D-2 


H 


49 


D-2 


A 


51 


D-2 


A 


54 


D-2 


A 


55 


D-2 


A 


60 


D-2 


A 


a 


D-2 


A 


63 


D-2 


A 


67 


D-2 


H 


69 


D-2 


A 


72 1 


D-2 


A 



Percent of Pupils 
Answering Correctly 
•at'l I School 



Phils. 



* 



3L 



-J3_ 
-31- 

3L 



Jk 



-22. 



72 



-51 



28 



J9_ 



I 



1 



s 

22 

Ji2_ 
JtP. 

J»k 
JZ. 

JtL 

-2JL 
50 

JUL 



70 



1 



J2- 
JlL 

JL. 
-2k 
-3k 

_ 2k 
-26- 



+ 2 



+11 



- 2 



j^k 



- 2 



- l 



- i 

- 5 

- 5 
-io* 

- 7* 

- 5 

-..85, 

- 9* 
-11* 

-11* 
-=15». 

-16* 



Nuober of Iteas 


- 31 


Sub of + Differences 


-+H 


Sua of - Differences 


--191 


Algebraic Sum 


■»-177 


iigeb£ejLc 


„SaP -172. =-6 


Average - 31 


31 



H - Hard; A - Average} K - Easy 



* Denotes Items having a difference between national and school percentages 
greater than the average of Item difficulty differences for the school. 
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Concl us ions 



The use of the test data for a variety of decision-making purposes in 
the School District of Philadelphia appears to provide testimony that an 
appropriate organization and production of test data reports can enlarge the 
vistas of users with respect to data applications. 

Educational Importance of the ftudy 

As researchers and practioners we need to explore new avenues for data 
use in the pursuit of ways to improve our instructional programs for our 
students. 

The complex context of education today demands improved decis i on-maki ng 
processes. We must, as a profession, respond well to those demands. 

The recommendations and complete discussion of the models described in 
this paper may provide catalytic help for others in their desire to improve 
test data utilization. 
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